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ABSTRACT 

This teacher's edition of training materials on 
bilingual program planning, implementation, and evaluation focuses on 
foundations for evaluating bilingual programs. The guide is part of a 
series directed at bilingual educators and intended for use in 
institutions of higher education and inservice teacher education 
programs. Training <ftbject ives , a pretest and posttest, suggested 
training procedures, and instructional activities and materials are 
included. Among the topics covered are (1) planning and organizing a 
bilingual program evaluation, (2) the evaluation team, (3) evaluation 
objectives, (4) establishing evaluation questions and methodologies, 
(5) collecting and ana3yzing evaluation evidence, and (6) using 
evaluation results. (HW) 
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BILINGUAL EDUCATION 
TEACHER TRAINING MATERIALS 



The bilingual education teacher training materials developed by the 

Center for the Development of Bilingual Curriculum - Dallas address five 

broad areas of need in the field of bilingual education: 

Series A: Bilingual Program Planning, Implementation, 
and Evaluation 

Series B: Language Proficiency Acquisition, Assessment, 
and Communicative Behavior 

Series C: Teaching Mathematics, Science, and Social 
Studies 

Series D: Teaching Listening, Speaking, Reading, and 
Writing 

Series E: Actualizing Parental Involvement 

These materials are intended for use in institutions of higher education, 
education service centers, and local school district in-service programs. 
They were developed by experts in the appropriate fields of bilingual educa- 
tion and teacher training. 

Series A addresses the critical issue of the effective planning and 
implementation of programs of bilingual education as well as efficient 
program evaluation. Sample evaluation instruments and indications for 
their use are included. Series B contains state-of-the-art information 
on theories and research concerning bilingual education, second language 
acquisition, and communicative competence as well as teaching models and 
assessment techniques reflecting these theories and research. In Series 
C, the content, methods, and materials for teaching effectively in the 
subject matter areas of mathematics, science, and social studies are pre- 
sented. Technical vocabulary is included as well as information on those 

vii 



aspects rarely dealt with in the monolingual content area course. 
Series D presents the content area of language arts, specifically the 
vital knowledge and skills for teaching listening, speaking, reading, 
and writing in the bilingual classroom. The content of Series E, totu,- 
alizin g Parental Involvement , is directed toward involving parents with 
the%Gtoool system and developing essential skills and knowledge for the 
decision-making process. 

Each packet of the series contains a Teacher Edition and a 
Student Edition. In general, the Teacher Edition includes objectives 
for the learning activity, prerequisites, suggested procedures, vo- 
cabulary or a glossary of bilingual terminology, a bibliography, and 
assessment instruments as well as all of the materials in the Student 
Edition. The materials for the student may be composed of assignments of 
readings, case studies, written reports, field work, or other pertinent 
content. Teaching strategies may include classroom observation, peer 
teaching, seminars, conferences, or micro-teaching sessions. 

The language used in each of the series is closely synchronized with 
specific objectives and client populations. The following chart illus- 
trates the areas of competencies, languages, and intended clientele. 



COK'UlT.CIfS, IANGJAGE Of INSTRUCTION ANT INTENDED CL !f NULL 



ARIAS OF COMPETENCIES 


LAN GUAM 


ClitNTuE 


SERIES A 6llingudl Prograr Planr-ing, 

Implementation, and Evaluation 


EnJ i$n 


Priranlj s*>pe r » iso^s 


SERIES B Lanquage Proficiency Acquistion, 

Assessment, and Communicative Behavior 


Spanish/ 
Engl ish 


Pnnarily teachers 
and supe^ isors 


SERICS C. Teaching Mathematics, Science, and 
Social Studies 


Spanish/ 
English 


Primarily teachers 
and paraprofessionals 


SERIES D leaching listening, Speaking, Reading, 
and Writing 


Spanish/ 
Engl ish 


Primarily teachers 
and Parap» ofessionaH 


SERIES i . Actualwing Parental Involvement 


Spanish 


Primarily teachers, 
parents, and ccxnnumtj 
1 laisons 
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In addition to the materials described, the Center has developed 
a Management System to be used in conjunction with the packets in the 
Series. Also available are four Practicums which include a take-home 
packet for the teacher trainee. 

The design of the materials provides for differing levels of lin- 
guistic proficiency in Spanish and for diversified levels of knowledge 
and academic preparation through the selection of assignments and strate- 
gies. A variety of methods of testing the information and skills taught 
in real or simulated situations is provided along with strategies that 
will allow the instructor to meet individual needs and learning styles. 
In general, the materials are adaptable as source materials for a topic 
or as supplements to other materials, texts, or syllabi. They provide 
a model that learners can emulate in their own classroom. It is hoped 
that teacher trainers will find the materials motivational and helpful 
in preparing better teachers for the bilingual classroom. 
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OBJECTIVES 

Upon the completion of this Packet, the student will be able to: 
Part I: 

1. Establish the evaluation roles of program staff, parents, community, 
students, and the external evaluator. 

2. Identify and establish working agreements with an external evaluator. 

3. Determine and establish the priorities of the evaluation objectives. 

4. Specify evaluation questions, monitor and/or specify information 
sources and instrumentation, the evaluation design, the analysis plans, 
and the contents of the evaluation report. 

Part II : 

5. Participate effectively and efficiently in the gathering and organ- 
ization of evaluation evidence. 

6. Monitor the analysis and interpretation of evaluation evidence. 

7. Monitor and participate i_n the reporting of the evaluation. 
Part III : 

8. Use evaluation results for program planning. 
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PRETEST 

Part I; 

Establishing an Evaluation Team 

1. What are the major activities of an evaluation? 

2. What are some of the ways in which parents, communi ty, and students 
may participate in a Title VII program evaluation? 

3. Name five important qualities that an evaluator should possess. 

4. Name five possible sources of information about potential external 
evaluators. 

5. What is a common pitfall in kicking off a program evaluation? 

6. Describe two ways to establish initial agreements with a program 
evaluator . 

7. What are some of the key elements to be included in the final agree- 
ments reached with the external evaluator? 

Determining the Purposes and Objectives of the Evaluation 

1. Name five possible purposes of program staff for conducting a 
program evaluation. 

2. In what two categories can evaluation objectives be placed? 
Describe the difference between these two basic approaches to 
program evaluation. 

De termining the Evaluation Questions, Va riables, and Methodology, 

1. Describe the sequence of events following the establishment of the 
final roster of evaluation objectives Which specify the elements 
of the evaluation design and process. 

2. Name and discuss two important considerations in the selection of 
language proficiency measures for the evaluation. 

3. What are some of the drawbacks of evaluation "burden" on staff and 
students? 

4. Typically, what will be contained in an evaluation report? 
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Part II : 

Gathering Evaluation Evidence 

1. What are some of the common concerns that may arise during the 
collection of evaluation evidence? 

Ana lysis of Evaluation Evidence 

2. How can program staff facilitate the preparation of evaluation 
evidence for analysis? 

Reporting the Evaluation 

1. What are the four basic types of reporting in an evaluation? 
Part III; 

1. What are the four basic things usually involved in usina evaluation 
results? 
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PRE-TEST ANSWERS 



Part I : 

E stablishing an Evaluation Team 

I. What are the major activities of an evaluation? (p. 9) 

What are some of the ways in which parents, community and students 
may participate in a Title VII program evaluation? tP- i«J 

Name five important qualities that an evaluator should possess, (p. 11-12) 

Name five possible sources of information about potential external 
evaluators. (p. 13) 

5. What is a common pitfall in kicking off a program evaluations (p. 13) 

6. Describe two ways to establish initial agreements, with a program 



2. 

3. 
4. 



7. 



evaluator. (p. 13-14) 

What are some of the key-elements to be included in the final agree- 
ts reached with the external evaluator? (p. 
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Determining the Purposes and Objectives of the Evaluation 

1. Name five possibL purposes of program staff for conducting a 
program evaluation, (p. 15) 

2. in what two categories can evaluation objectives be placed? 
Describe the difference between these two basic approaches to 
program evaluation • (p- 34) 

Determining the Evaluation Questions, Variables, and Methodolo^ 

1. Describe the sequence of events following the establishment of the 
final roster of evaluation objectives Which specify the elements 
of the evaluation design and process, (p. 17) 
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2. Name and discuss two important considerations in the selection of 
language proficiency measures for the evaluation. tp. W 

3. What are some of the drawbacks of evaluation "burden" on staff and 
students? (p. 20) 

4. Typically, what will be contained in an evaluation report? (p. 40) 
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Part li s 

Gathering Evaluation Evidence 

1. What are some of the common concerns that may arise during the 
collection of evaluation evidence? (p. 23) 

Analysis of Evaluation Evidence 

2. How can program staff facilitate the preparation of evaluation 
evidence for analysis? (p. 24) 

Reporting the Evaluation 

1. What are the four basic types of reporting in an evaluation? (p. 25) 
Part III; 

1. What are the four basic things usually involved in using evaluation 
results? (p. 27) 
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ACTIVITIES 



Session 



1. Review objectives for entire Packet. 

2. Administer pretest. 

3. Present Part I, A: 

o discuss the content of each section; 
o review Exhibit I and II; 

o role play the initial conference. with an external 
e valuator ; 

o discuss problems which the participants have experienced 
o^observed* in arrangements between a Title VII program 
and either internal or external evaluators. Discuss 
ways to prevent or resolve these. 

Assignment: 

o Obtain copies of several different program evaluations. 
Prepare to present each of these to the class in small 
groups of two to four people . 

o Read Part I and II. 

1. Provide time for small groups to prepare their presentations 
on various program evaluations. Included in the presentation 
should be : 

o specification of **the program objectives; 
o specification of the evaluation objectives ; 
o specification of the priorities among objectives; 
o distinction between process and product evalution 
objectives; 

o specification of the evaluation questions: 
o specification ofr information sources, instrumentation, 
'and methods. 

2. Present Part I. B & C. 

o discuss the content of each section; 
o review Exhibit IV and V 

o discuss the problems involved in presenting technical 
statistical data on evalutions to the audiences that 
will be interested in the evaluation. What approaches 
to statistical information are particularly helpful 
to program staff. Have good examples been found in 
the evaluations that participants are preparing for 
presentation? What are some examples of bad approaches? 

o discuss how hypothetical tables of results such as in 
Exhibit VI can be developed prior to the analyses. 
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Session 



1. Discuss the contents of an evaluation report, what kind of 
information and formatting does a program staff find partic- 
ularly useful. Is there a process for developing the report 
that can be recommended? 



2. Presentations of evaluations by groups of participants. Examples 
of exemplary approaches should be presented. Problems should 
be brought to the attention of the class and ideas for how 
to prevent or resplve these discussed. Note should be taken of 
how different program evaluations address process versus -product 
evaluation questions. What was most informative about these 
evaluation reports? What was the relative cost of the various 
evaluations? How were these differences in cost revealed in the 
nature and quality of the evaluation? How would the presenting 
group suggest revising the evaluation to make it more useful 
and more cost-effective? 

Assignment: 

o Read Part II and III 



5, 6, 7 1. Present Part II and III 

2. Have participants break up into working groups again. 

Have each group develop a questionnaire and an interviewing 
procedure for obtaining evaluation information from program 
staff on process evaluation questions such as staff training, 
implementation of an innovative curriculum, parent and 
community training. 

3. Have participants pretest the questionnaire and interviewing 
procedure on friends and colleagues in order to work out the 
problems and bring it to final form. Participants not familiar 
with survey instrument development should refer to the Weisberg 
& Bowen reference, or to one of the many other texts on survey 
research or questionnaire development. 

4. Each participant group should provide copies of the questionnaire 
to each other member of the class and, if possible, to other 
individuals as well. Each group should also use the interviews 
with two or three individuals. They should then summarise the 
results of the questionnaires and interviews in tabular, graphic, 
or other recommended formats and present the results to the class. 



15 

8 

ERIC 



PART I: PLANNING AND ORGANIZING THE EVALUATION OF A 
TITLE VII PROGRAM 



Establishing an Evaluation Team 

Identifying Evaluation Roles of Program Staff. The evaluation 
of a Title VII program may be conducted by an internal evaluator, 
an external evaluator, or an evaluation team which combines internal 
and external evaluation resources. In most cases, program staff, 
parents, community members, diltrict and school board representatives, 
and students will all play seme role in the evaluation. Because 
the time and energies that any one of these groups has to devote to 
evaluation is scarce, it is essential that their evaluation respon- 
sibilities be defined very clearly and not exceed the amount of time 
they will actually have available. This is particularly true of pro- 
gram staff who have many other responsibilities. 

The major activities of an evaluation include planning, 
collecting information, analyzing the information, interpreting 
this information into conclusions and recommendations, reporting 
the results, planning ways to use the results, as well as monitoring 
the evaluation process. If the evaluation is being conducted 
internally, the program staff and other program participants will 
be involved in all of these activities. If major responsibility 
is given to an external evaluator, the program staff will most 
likely be involved in planning, collecting information, and monitoring 
the evaluation process. In either case, the total scope of staff 
and evaluation activity should be fully described, with each task and 
responsibility explicitly assigned to individuals, and a firm schedule 
for the completion of each task. This process may begin in the early 
planning stages as a tentative sketch of the evaluation roles and 
time allocations of program staff and external evaluator. But by the 
time final decisions are being made on the objectives of the evaluation, 
the methodology, and the working relationship with an external eval- 
uator, there should be a detailed mutual understanding of the respon- 
sibilities of program staff for the evaluation. 
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Identifying Evaluation Roles of Parents , Community, and Students , 
Parents , students , and community members may often 'play important 
advisory and participatory roles in program evaluation. Their input is 
particularly important when the objectives and potential uses of the 
evaluation are being deliberated. They also have unique insights 
into the problems or facilitating factors which influence the imple- 
mentation of a Title VII program. Their perspective is invaluable 
when instruments are being reviewed for potential bias, or being pilot 
tested in order to eliminate ambiguous questions or to improve clarity. 
Their input should be sought on the question of test burden when the 
number of tests and the timing of testing is being considered. 



In sane cases, parent/community committees have been willing to 
assume an ambitious level of responsibility for some specific technical 
facet of the evaluation such as systematic classroom observation. 
Involvement in an activity such as this requires training, supervision, 
and a substantial time commitment. If undertaken , there should be 
a clear and shared understanding of the overall time, resource , and 
responsibility ccmmitment entailed, and an agreement to fulfill this 
commitment from beginning to end. 
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Although there are many fewer Title VII programs at the secondary 
level, critical educational issues exist at this level. Moreover, 
students may be the best sources of information on many of these issues. 
For example, the way in which the Title VII program is coordinated 
with career education and with minimum competence testing will be 
important to secondary students. Over 30 states currently mandate 
minimum competency tests for secondary students. There is evidence 
that language minority students are experiencing a disproportionately 
severe impact from some of these programs (e^g., California State 
Department of Education, 1981). Issues of educational goals and 
program approaches can be understood by secondary students, and 
they and their families will frequently have creative suggestions 
to make on how effective programming decisions may be made and 
implemented. Their contributions should now be forgotten in 
evaluation. 
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Identifying an Evaluator. If the decision has betsn 
made to employ an external evaluator, steps should be taken early in 
the program planning process to identify one or more likely indi- 
viduals or teams of external evaluatois to conduct tne evaluation. 
Ideally, this would be done during the proposal-writing stage. 
In this way, the objectives of the program and of the evaluation 
could be formed in a completely compatible way. The information 
needed for an effective evaluation could be designed into the ongoing 
program process. Moreover, the evaluation could be designed to 
address the priorities of the program and to fit the scope and budget 
of the program. 

Regardless of whether the task is to identify an internal or 
external evaluator, it is essential that the person or team chosen 
meet several key criteria. Perhaps first and foremost is the need 
for the evaluator to understand the philosophy and goals of bilingual 
education. This is usually apparent from a review of the individual's 
experience with Title VII programs or with other language minority 
programs in the past. Title VII program evaluation entails the entire 
range of technical, management, and interpersonal professional skills 
that are required by any educational program evaluation. However, 
it also requires special knowledge in the area of language assessment 
and bilingual education curriculum models. The potential evaluators 
should be interviewed in depth to assess their possession of these 
attributes. Some of the specific qualities or experiences that would 
be necessary include: 

1. Skill in organizing concepts and activities in a way that 
will result in a coherent and efficient evaluation plan; 

2. Skill in evaluation design and research design; 

3. skill in instrument selection and development for use 
with adults; 

4 Skill in instrunent selection and development for use 
with students, particularly with respect to language 
proficiency assessment (speaking, understanding, reading, 
writing) ; 
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5. Skill in classroom observation techniques; 

6. Skill in interviewing techniques; 

7. Skill in sampling techniques, and both qualitative 
and quantitative data analysis; 

8. Skill and experience in data management; 

9. Writing and public presentation skills; 
10. Project management skill and experience. 



In addition to these technical skills, the program team should £ 
feel comfortable that a productive interpersonal working relationship 
can be established with the evaluation personnel. This would require, 
at a minimum, satisfaction with the evaluator' s skill with human 
relations and personal communication, with the evalua tor's ability 
to inspire trust and to relate to the program's community with 
sensitivity, while maintaining objectivity. It is not enough for 
an e valuator to be able to spot problems in a program. The most 
appropriate commitment for an evaluator is to identify both problems 
and facilitating factors, and to communicate these in an ongoing 
manner to program staff along with recommendations for realistic 
approaches to solving problems. Another consideration in the 
selection* of an evaluator might be the extent to which the candidate 
is able and willing to provide training to others who are to be 
involved in the evaluation* A key consideration is whether or 
not the potential evaluator is able to make a firm commitment to 
devote the amount of time and attention necessary to satisfactorily 
address the evaluation questions. In order to stiucture this 
commitment, it will be necessary for the program staff to be very 
clear in the number of days that will be necessary to perform the 
evaluation, when the work will occur, the degree of support that 
the program staff and other program role players will provide to 
the evaluator, the amount of money available to compensate the 
evaluator, and the schedule of payment to be employed*. 

There are many sources of information about potential external 
program evaluators. Bissel (1980) has compiled a useful list of 
sources: 
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o Federal project officers; 

o State education agencies, particularly offices of bilingual 
education, program evaluation, or research; 

o colleagues who may have names of potential candidates, 

and people in other districts responsible for similar programs; 

o Authors of evaluation reports, journal articles, or other 
publications which reflect Knowledge of salient issues; 

o People who have given presentations applicable to project 
needs at conferences and workshops; 

o The ERIC microfiche system (which is available at most 
university libraries) which has pertinent evaluation 
reports; | 

o The evaluation centers, education, or psychology departments 
of nearby colleges or universities; 

o Independent research organizations and consulting firms 
having expertise in bilingual education evaluation. 



Initial Conference with An External Evaluator . There are 
countless appropriate ways to kick off a program evaluation. However, 
one common pitfall is a prolonged and unsure beginning, marked by 
unclear canmunication and vague agreements between the staff and the 
evaluation team. This can be very costly in terms of professional 
time. Since most Title VII evaluation budgets are modest, it is 
important that professional time be used as efficiently as possible. 
One way of doing this is to schedule an initial conference with 
the potential evaluator (s) to establish the conceptual scheme 
and basic under standings of the evaluation. The result of this 
meeting should be either a tentative agreement to proceed in specific 
ways, or an agreement that the candidate will submit a final plan 
which the program will review for a final decision. 

In advance of the initial conference with the one or more most 
serious evaluator candidates, each should be provided with basic 
information about the Title VII program. This should include the 
program^ objectives, evaluation ideas presented in the original 
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proposal, information on the staff's evaluation priorities, 



deadlines for various critical phases of the evaluation, the 
evaluation budget, and information on the extent to which the 
evaluation can expect assistance and resources from the program. 
The potential evaluators should be instructed to review these 
materials carefully and to be prepared to give a conceptual 
overview of how they will conduct the evaluation. They should be 
reacSy to ask questions about program models and instructional 
approaches, classroom and program configurations, numbers of students, 
control groups, and other matters necessary to their final evaluation 
formulation. 

Program staff should review the candidate's resume and references 
prior to the meeting. This will permit the staff to use this conference 
to explore the candidate's understanding of the goals of bilingual 
education in general, the program's objectives in particular, and the 
candidate's past experience with skills needed in Title VII evaluations. 

Another approach to establishing these initial agreements is to 
issue a request for proposals to a list of potential evaluators. If 
this approach is used, a Request For Proposal (RFP) document will have 
to be prepared and formally distributed. A set of objective criteria 
for judging the relative merits of the proposals and a review committee 
will have to be established. The time consuming nature of this 
approach may be prohibitively expensive in both financial and programmatic 
terms. It is frequently more productive and less expensive for 
the program to follow the more informal process of locating a list of 
candidates and conferring with them. Even with this approach, however, 
there should be written criteria, a clear process, and a decision- 
making timetable established and shared with the candidates. 

Final Agreements . Once the program stafVf and the chosen evaluator 
have reached final agreements on the scope and nature of evaluation objec- 
tives and activities, budget, and schedule, these matters should be de- 
scribed in a written form and signed by both parties. This contract con- 
stitutes the formal arrangements for the evaluation process and product. 
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Determining the Purposes and Objectives of t he Evaluation 

Specifying the Requirements of the Funding Agency . The Office 
of Bilingual Education and Minority Languages Affairs (OBEMLA) uses 
evaluations for the purpose of being informed about local projects. 
The evaluation reports help them establish priorities for site 
visits and to judge how successful projects are in meeting their 
objectives. Therefore, it is important to consult OBEMLA as a first 
step in deciding how to structure and rank program evaluation objec- 
tives. 

Specifying the Priorities of the LEA and Program Staff . Project 
staff are likely to place emphasis on measuring student progress, deter- 
mining whether specific objectives are being met, and obtaining ongoing 
feedback that will facilitate the management and possible modification 
of the program. As Bissel (1980) notes, other staff interests might be 
how to decide the most effective approaches to use with different types 
of students; to determine how the project activities could be improved; 
to identify learning patterns -of students with different levels of 
speaking, understanding, reading, or writing proficiency in English and 
their first language; to look at the difference in rates of instruc- 
tional progress of students in different projects; to gain and maintain 
school board and community support for the project. 

The school board may likely be interested in many of these same 
matters. It may also be interested in the cost effectiveness of the 
program, and the feasibility of continuing the program once funding 
has expired. Parents and the community representatives will probably 
expect that the evaluation will provide them with basic information 
about both the implementation of the program as planned and how 
effective it has been in terms of student performance. 

Defining Evaluation Objectives . This is the first step in 
establishing the evaluation design. The evaluation objectives will 
provide the framework for the evaluation questions that will be 
asked and the methodology that will provide answers to those questions 
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by collecting, analyzing, and interpreting information. Initially, 
what is needed is a listing of all potential evaluation objectives. 
Most of these items will reflect either process or product program 
objectives. Process objectives are concerned with the implementation 
of the Title VII program. Typically, these would include items such 
as staff and parent training, curriculum and material development and 
identification, coordination with district/parent/conmunity repre- 
sentatives, and program planning and administration functions. Product 
objectives are typically concerned with student performance and other 
outcomes. Thus, most evaluation objectives will address the evaluation 
of the key program objectives . In addition, there should be an eval- 
uation objective that deals with the reporting of the evaluation results. 

Ranking Evaluation Priorities . Because the interests of various 
role players in any Title VII program have somewhat different emphases, 
there may be more evaluation objectives suggested than are feasible to 
address with the evaluation resources available. It is therefore 
crucial that clear priorities among these objectives be established. 
The priorities of the funding agency must be addressed if continued 
program funding is desired. The priorities of program staff should 
represent those objectives most necessary to the operation of the 
program in a productive and cost-effective manner. Perhaps the most 
important activity during this ranking process is to sift the 
essential from the interesting-but-not-essential , and f ran the trivial . 
Once these priorities have been established, they will provide the 
decisionmaking framework for many subsequent evaluation decisions. 
For example, a questionnaire or interview guide for staff or other 
program role players should address only topics crucial to the eval- 
uation objectives. Additional irrelevant questions should be deleted* 
In other words, establishing the evaluation objectives and the priorities 
among them establishes the allocation of evaluation resources. 
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Determining the Evaluation Questions, Variab les, and Methodology ~~ 

Specifying Evaluation Questions . After the final roster of 
evaluation objectives has been established and ranked, each objec- 
tive should be further detailed into its component evaluation 
questions. Each question should be separate and non-overlapping 
with each other. When combined, these questions should cover all 
relevant and priority aspects of each evaluation objective. An 
examination of the sample evaluation questions specified for the 
sample product and process evaluation objectives (Exhibit IV, P. 36) 
shows how each question carefully queries the relationship among 
certain key variables. For example, the first evaluation question 
which concerns English oral language development sets out English 
oral language proficiency as the dependent variable. It establishes 
pre-program performance as the occasion of interest, and defines 
the population of interest as LEP students at each grade in the K-6 
grade range, in both the program and comparison groups. These 
specifications set the stage for the next logical development in 
the evaluation: the specification of measurable or identifiable 
variables or indicators which will answer each question, specifica- 
tion of the sources of this information, and identification of the 
instruments and methods to be used to obtain this information. 

S pecifying Information Sources, instru mentation, and Methods. 
Exhibit V, p. 38 uses the first evaluation objective of Exhibit III, p. 34 
and the corresponding evaluation questions (shown in Exhibit IV, p. 36) as 
the basis for showing how a plan may be developed to obtain the answer 
o each evaluation question. For each evaluation question, the exact na- 
ture of the data needed to answer the question is defined. Secondly, a 
decision is made upon the best and most cost-feasible source of this 
data. Decisions regarding the data source raise questions about the 
scope of data collection, and about sampling issues. For example, 
if the budget of the evaluation permits, it would be wise to 
obtain information not only from the program classroom on dif- 
erent occasions, but also from one or more comparison groups 
which differ in specific and documentable ways from the program 
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classrooms. If the budget permits, student data would probably 
toe gathered on each child in each classroom to be included in the 
evaluation. However, if the number of students is large and/or 
the evaluation budget is small, savings could be achieved by obtaining 
a random sample of students. If a sample is decided upon, the 
exact nature of the sampling rationale end procedures should be 
specified to ensure that the sample is an adequate representation 
of % the entire population. 

The final column of Exhibit V, p. 38 deals with the instrument 
or method of obtaining the information needed to answer the evaluation 
questions. The example presented requires decisions regarding the 
assessment of oral language proficiency - a difficult consideration. 
Although there are many instruments purporting to measure- the 
English oral language proficiency of students, few have evidence 
of validity and reliability. It is very important that project staff 
as well as the external evaluator becone knowledgeable about the 
relative psychometric merits of the instruments being considered. 
If the program and evaluation objectives address the development and 
assessment of first language proficiency, even greater caution is 
required in the selection of measures, as measures of non-English 
languages have an even less adequate research history and those 
which provide measures of both English and another language 
can seldom be considered comparable (Merino and Spencer, 1981) . 
Similar problems arise when the measurement of reading and writing 
i& at issue. Selected readings on this subject are cited in the 
bibliography, and it is covered in depth in Packet 2 of this series. 
Fxogram staff will also want to consider the potential for making 
multiple uses of the instruments chosen for the evaluation. If an 
instrument may appropriately be used for a combination of functions 
(e.g., student classification, placement, diagnosis, and evaluation), 
money may be saved in the purchase, administration, and analysis of 
test results. In addition, the student will be relieved of some 
test burden. However, many tests are not appropriate for some of 
these functions, and if used in these ways may be damaging to both 
the students and the program. 
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In order to obtain answers to questions regarding classroom 
practices, it is usually necessary to both observe a sample of classroom 
time and to ask questions of teachers. A large number of classroom 
observation instruments are available for this purpose (e.g. 
Simon and Boyer, 1965) . In recent years, sane have been developed 
which are appropriate to the observation of language development . 
techniques and context variables (e.g., Marliave, Fisher, Filby, and 
Dishaw, 1980; Southwest Educational Development Laboratory, 1978) . 
All such instruments require training that prepares observers to 
make reliable observations, scoring, and interpretive judgments. 
The time-consuming nature of these measures usually makes them most 
appropriate for large Title VII programs, particularly those attempting 
to test innovative techniques and those with substantial research and 
evaluation budgets. 

In order to obtain answers to most process questions, it will 
be necessary to either interview program staff or to have them 
complete self-administered questionnaires. The program evaluator 
should have experience in constructing questionnaires and interview 
guides for this purpose. The questions asked of staff and other 
program participants should directly reflect the evaluation questions 
and not stray from these concerns into other matters. Each such, 
irrelevant question creates unnecessary administration, analysis and 
staff costs. The structure and wording of these instruments should 
be carefully checked by the program staff supervising the evaluation 
to ensure that it will provide clear and unambiguous answers to the 
evaluation questions, and that the evaluator has given full consid- 
eration to how the questionnaire and interview responses will be 
coded and used in a specific analytic approach. It is very useful 
at this point to see a mock-up of the tables and graphs that will 
result from the information yielded by the instruments. These 
often will make it clear just how well the information being collected 
will answer each evaluation question. 

Once specifications such as those in Exhibit V, p. 38 are available 
for each evaluation question, the program staff should consider the 
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evaluation burden that this plan implies for the various role players 
in the program. Are the students being tested so extensively that 
their learning time is being eroded? Are teachers and program staff 
expected to spend so much of their time on evaluation interviews,, & 
questionnaires, miscellaneous paperwork, and activities that their 
program activities are hindered? When the answers to these questions 
are yes , then the evaluation burden on these role players must be 
trimmed back or scheduled in more convenient ways. The potential 
for these burdens and the resulting frictions or program disadvantages 
often do not become apparent until it is too late unless some process 
such as the definitional one displayed in Exhibit V, p. 38 is used to 
clarify the situation. 

Specifying Evaluation Designs and Analysis Plans . Most evaluations 
will include both quantitative and qualitative information. The 
evaluation design and analysis plan will treat these two types of 
data differently. The example used in Exhibit V, p. 38 is concerned 
with data which is quantifiable and which can be analyzed statis- 
tically to determine the answers to the research questions. The 
evaluator should be able to clearly describe to program staff what 
these technical procedures are and how they will produce answers to 
the evaluation questions. One of the best ways to do this is to 
provide hypothetical samples of how the major tables and graphs 
will look once these answers are in. Along with each of these, 
the evaluator will be able to note which statistical analyses 
will be performed, if any. The use of descriptive statistics 
such as percentages, percentiles, means, and standard deviations, 
as well as graphic displays are frequently very useful to program 
staff who must understand the results of the evaluation in order 
to use them to improve the program. Exhibit VI in page 40 
provide* examples of how a display of hypothetical results can set 
forth analytic procedures to be used and provide program staff 
with a preview of what they can expect from the evaluation report. 

Analyzing information on process objectives usually requires 
a qualitative approach. Although the questions and interviews may 
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yield some numerical ratings, most of the information obtained 
will be of a judgmental nature or will identify critical 'incidents 
or conditions. The evaluator must examine this information and 
organize .<t in a way that identifies and summarizes dominant 
themes and explains their significance and program implications. 
Exhibit VII, p.Al displays a"format that is useful for the organizing 
of process information in this way. 

Specifying the Contents of the Evaluat ion Report. Once the struc- 
ture of the evaluation has been established in the preceding tasks, the 
structure and content of the evaluation report have basically been 
established. Typically, the evaluation report will describe the proj- 
ect, provide information on the evaluation methodology, report the 
results of the evaluation of both product and process evaluation 
objectives, draw conclusions and make recommendations. The table 
of contents will therefore resemble that shown in the following chart. 





TITLE VII PROGRAM EVALUATION 




* Table of Consents 


I. 


Project Inscription 


II. 


Evaluation of Instructional Objectives 




o Methodology 




o Outcomes 


III. 


Evaluation of Implementation Objectives 




o Methodology 




o Outcomes 


IV. 


Conclusions! 




Rec GUtme nd a t/i on s 
/ . . — . — 
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The text in each outcome section should systematically answer 
each evaluation question. For the quantitative evaluation 
questions, these answers should include both narrative and tabular 
or graphic explanations. This narrative should be a clear and simple 
explanation of the results shown in the tables and graphs planned 
in advance to answer the evaluation questions. The answers to 
qualitative process questions may well follow the format of Exhibit 
VII, p. 41 providing information on intended and actual per- 
formance, as well as important issues which either inhibit or 
facilitate the accomplishment of each process objective. 

At the point that the evaluation has reached tentative conclu- 
sions and recommendations, it is time for the program staff respon- 
sible for coordinating the evaluation to have a preview of the report. 
Their input should be sought on both conclusions to be drawn, and on 
recommendations to be made. By viewing the results once they are 
organized for reporting purposes, program staff are frequently able 
to, make a further analysis of important factors which influenced the 
program during the year, and to suggest ways to proceed in the future. 
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PART II: IMPLEMENTING THE EVALUATION OF A TITLE VII 
PROGRAM 

Gathering Evaluation Evidence 

The process of gathering evaluation data usually consists 
of the following five steps: 

1. Orienting program staff to the data collection procedures 
and schedule that will be used, and modifying these 
where appropriate; 

2. Training participating staff in the administration of student 
measures; 

3. Scheduling and conducting interviews ; 

4. Scheduling and administering student performance measures; 

5. Coordinating and monitoring interviewing, observing, and 
te sting. 

The program evaluator will implement these steps. Program 
staff responsible for supervising the evaluation should be available 
to help solve scheduling and coordinating problems, and to monitor 
the quality of training, testing, and interviewing activities. Prob- 
lems that arise during this phase frequently revolve around timing. 
External evaluators are often unfamiliar with school holidays, minimum 
days, certain overload periods such as those around report card time, 
and daily class schedules. The evaluator must depend upon the program 
staff to provide advance guidance on these scheduling matters in order 
that data may be obtained in a way that protects its reliability 
and validity, while simultaneously ensuring that all data is collected 
at the proper point in time. In a similar way, evaluators will need 
guidance on the line of authority in the project in order to know 
exactly whom to relate to on each major concern. 

Another matter of concern during this time is the organization and 
security of data collection materials. The logistics of transporting 
and storing student testing materials may require the assistance and 
consideration of program staff. The same is true of the evaluators 
need for private quiet space for individual testing and interviewing. 

23 

30 



The evaluator or other members of the evaluation team may not 
speak all of the languages represented by the Title VII program 
fluently. If this is the case, it will be necessary for program 
staff who do speak these languages to accompany the evaluator 
during presentations and interviews where the use of these 
languages would be either beneficial or necessary. 

Preparing Evaluation Evidence for Analysis 

) 

Program staff can 'frequently contribute to this activity in 
ways that reduce the time (and therefore the cost) of the evaluation. 
A system should be devised for recording student demographic, 
testing, and program information in a centralized manner. If this 
information is conscientiously recorded into a suitable format when 
first obtai r ^ and transmitted to the program office, vast amounts 
of duplicated recording work can be avoided. Exhibit II, p. 31 displays 
an example of how one Title VII program office did this. This excellent 
example of record keeping was given directly to a key punch operator 
who then created the computer cards used in the analysis of student 
data. Thus, by careful advance planning and coordination by the pro- 
gram staff and evaluator, it was. possible to skip the whole step of 
recording student data on keypunch forms. This process would have been 
equally useful if the evaluator had decided to analyze the data manu- 
ally. 

Analysis of Evaluation Evidence 

Decisions regarding the specific analytic approaches to be 
used to obtain answers to the evaluation questions should be made 
in. the planning phase of the evaluation. They should not be left 
to be made in the later phases of the evaluation. Otherwise, the, 
data collected^may not be amenable to appropriate means of analysis. 
Thus, it should be decided early not only which analyses will be 
made, but whether a computer or manual analysis will be applied 
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to quantitative data. The costs and scheduling of analysis should 
be clearly and realistically projected. It is unlikely that analysis 
can be accomplished quickly. At least a month should be allowed 
for this function in most Title VII evaluations. Analysis includes 
not only the completion of descriptive and inferential statistical 
analyses, but interpretation of results as well. The development of 
written interpretations constitutes the core of the final evaluation 
report. 

If possible, preliminary analysis of key evaluation questions 
should be scheduled to provide formative evaluation results. That 
is, information on the implementation of certain key program activities 
may be obtained and analyzed early in the program year in order to 
provide the program with feedback in an ongoing manner. This will 
allow the development of "on-course" corrections that may possibly 
bring the program closer to achieving its goals. 

Reporting the Evaluation 

Basically four types of reporting are desirable in an evaluation: 
1) periodic reports containing preplanned formative evaluation 
information of either a process or product nature; 2) a preliminary 
version of the final report for review and input by the program 
staff; 3) a final version of the final report, accompanied by a 
nontechnical "executive" summary; and 4) an oral presentation of 
evaluation findings and implications to program staff, district/ 
parent/community representatives. This latter type of report is 
often neglected, but it has the potential of setting the stage for 
the most productive use of evaluation results. It may be appropriate 
to use it as the springboard for modifying or extending the program 
objectives, or for planning new program proposals. It will be most 
useful if information on the evaluation is available to interested 
parties in advance of the meeting, and if a participatory format, is 
used. 
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PART III: USING EVALUATION RESULTS 



Four basic things are usually involved in the utilization 
of evaluation results: 

1. Identifying corrective actions to address evaluation 
findings and recommendations; 

2. Making revisions in the program in response to evaluation 
findings and recommendations; 

3. Evaluating the effects of revisions and modifying them 
where indicated; 

4. Disseminating evaluation results to individuals and groups 
concerned with the program, as well as to educators and 
interested audiences in general. 

Too often, program evaluations are considered bothersome - 
necessities which are performed in a perfunctory manner. But to the 
contrary, if planned carefully so that they conform to a feasible 
scope of work for both the program and the evaluator, they may become 
a valuable resource to the entire programming process. If this is 
to be the case, the questions asked by the evaluation should be of 
importance and interest to the program. This should not be difficult 
considering the current state of knowledge about educating language 
minority students. Even more importantly, the results of program eval- 
uations should be used. What if it is found that the training of staff 
did not provide them with adequate skills to implement the instructional 
techniques? What if secondary students report that the bilingual program 
is irrelevant to their career needs? What if two of the five program 
classrooms produced amazing performance gains while the others had mediocre 
results? What if the results were inconclusive or confusing? The program 
should not ignore any such findings. Each should be considered carefully 
by all program role players and used as a building block for the next 
educational occasion. 
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EXHIBIT I 



EXAMPLES OF SELECTED EVALUATION ACTIVITIES 



EVALUATION ACTIVITIES SCHEDULE 



EVALUATION ACTIVITY 



PERSON 
RESPONSIBLE 



DATE OF 
Initiation Completion 



COIWENTS 



0.0 Development of Evaluation Related 
Components of Application 

0.1 Assist In conducting needs 
assessment. 

0.2 Work with project staff, 

parents, school district ad- 
ministration, school board 
and students In developing 
general evaluation approach. 

0.3 Write evaluation section of 
application in conjunction 
with a team of project staff 
and participants. 



2.0 Evaluation Planning 

2.1 Determine specific informa- 
tion necessary to address 
evaluation audiences, pur- 
poses and questions. 

2.2 Identify specific evaluation 
participants and evaluation 
monitoring team. 

2.3 Determine needs of project 
staff, parents and community, 
or arrange for its provision. 
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EXHIBIT I 



EXAMPLES OF SELECTED EVALUATION ACTIVITIES 



EVALUATION ACTIVITIES SCHEDULE 



EVALUATION ACTIVITY 



PERSON 
RESPONSIBLE 



DATE OF 
Initiation Completion 



COMMENTS 



3.0 Designing the Evaluation 

3.1 Identify procedures for 
assessing attainment for 
each specific objective. 

3.2 Develop surveys, question- 
naires and other instrument: 

3.3 Determine data analysis 
methods • 



4.0 Measurement of Project Implemen- 
tation. 



(Steps for erch evaluation 
component should be 
designated .) 



7.0 Utilization of Evaluation Findings 

7.1 Identify corrective actions 
necessary to address evalua- 
tion findings. 

7.2 Revise project plans to in- 
clude corrective actions. 

7.3 Review progress in implement- 
ing corrective actions. 
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EXHIBIT IT 



Project Activities 



Project Months 



2. 



Conduct preliminary 
eval uat ion planning . 

Finalize purposes and ob- 
jectives of the evaluation 



3. Determine evaluation ques- 
tions, variables and meth- 
odology. 

4* Assemble and organize data 
and information needed to 
evaluate each program ob- 
jectives . 

5. Analyze student language 
proficiency and achievement 

■ data. 

6. Analyze and interpret eval- 
uation data on staff devel- 
opment activities. 

7. Prepare draft of final 
report . 

8. Confer with program staff 
about final report, 

9. Prepare final version of 
final report and executive 
summary . 



SEP 



OCT 



NOV 



DEC 



JAN 



FEB 



MAR 



APR 



MAR 



JUN 



JUL 



AUG 
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EXHIBIT II (CONTINUED) 





TITLE VII PROGRAM EVALUATION 
Person Loading Chart 


EVALUATION 
DIRECTOR 


EVALUATION 
ASSOCIATE 


UlCH 
>TANT 


(J pi 








CO to 
W CO 
rv* *i* 


CLERj 
SUPP( 




1. 


Conduct preliminary evaluation planning. 


2 


2 








2. 


Finalize purposes and objectives of the 
evaluation. 


1 


1 








3. 


Determine evaluation questions, variables 
and methodology. 












4. 


Assemble and organize data and information 














needed to evaluate each program objective. 


2 


5 


c 
J 






5. 


Analyze student language proficiency and 
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2 
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Analyze and interpret evaluation data on 
staff development activities. 




8 








7. 


Prepare draft on final report. 


3 


5 




3' 




8. 


Confer with program staff about final 
report . 


2 


2 








9. 


Prepare final version of final report and 














executive summary. 


3 


5 
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EXHIBIT II (CONTINUED) 



I . PERSONNEL 



Evaluation Director 
Evaluation Associate 
Clerical Support 



EVALUATION BUDGET 



Person 
Days 

12 

37 
6 



II. FRINGE BENEFITS 
III. OTHER DIRECT COSTS 

Research Associate (7 days (s $60.00 each) 
Keypunching 
Computer Analysis 
IV. INDIRECT COSTS «? .AO of I-TIl) 

' TOTAL COSTS 



Daily 
Rate 

$100.00 

75.00 

50.00 



Amount 

$1,200.00 
2,775.00 
300.00 

$4,275.00 
641.00 

420.00 
300.00 
500.00 
$2,454.00 

$8,590.00 
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EXHIBIT III 

CORRESPONDING PROGRAM AND EVALUATION OBJECTIVES 
SELECTED PRODUCT OBJECTIVES 



PROGRAM OBJECTIVES 



EVALUATION OBJECTIVES 



SPEAKING AND UNDERSTANDING IN 
ENGLISH AS A SECOND LANGUAGE 
FOR LEP STUDENTS . To provide 
individualized oral English 
language instruction to LEP 
students in grades K-6, which 
leads to significant improve- 
ment in their English oral 
language proficiency skills. 



To document and describe the 
extent to which LEP students 
in grades K-6 who receive 
individualized oral English 
language instruction improve 
their English oral proficiency 
skills each program year relative 
to a comparison group that does 
not receive instruction. 



READING IN ENGLISH AS A SECOND 
LANGUAGE FOR THE LEP STUDENT . 
To provide individualized 
instruction in the English 
reading continuum to LEP 
students meeting readiness 
criteria, which leads to 
significant improvement in 
their English Reading skills. 



To document and describe the 
extent to which LEP students 
who receive individualized 
instruction in the English 
reading continuum improve 
their English reading skills 
each program year relative to 
comparison groups receiving 
different forms of instruction 
or who vary in language back- 
ground characteristics. 



SPEAKING AND UNDERSTANDING IN 
SPANISH AS A FIRST LANGUAGE 
FOR LEP STUDENTS . To provide 
individualized oral Spanish 
language instruction to LEP 
students in grades K-6 whose 
first language is Spanish, 
which leads to significant 
improvement in their Spanish 
oral language proficiency skills. 



To document and describe the 
extent to which LEP students 
in grades K-6 who receive 
individualized oral Spanish 
language instruction improve 
their Spanish oral proficiency 
skills each program year 
relative to a comparison group 
that does not receive instruction. 
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EXHIBIT III (CONTINUED) 



,33 



PROGRAM OBJECTIVES 



EVALUATION OBJECTIVES 



READING IK SPANISH AS A FIRST 
LANGUAGE FOR THE LEP STUDENT. 
To provide individualized 
instruction in the Spanish 
reading continuum to LEP 
students meeting readiness 
criteria and whose first language 
is Spanish , which leads to sig- 
nificant improvement in their 
Spanish Reading skills. 



To document and describe the 
extent to which LEP students 
who receive individualized 
instruction in the Spanish 
reading continuum improve 
their Spanish reading skills 
each program year relative to 
comparison groups receiving 
no instruction or different 
forms of instruction • 



SELECTED PROCESS OBJECTIVES 



PROGRAM OBJECTIVES 



EVALUATION OBJECTIVES 



STAFF DEVELOPMENT . To 
provide preservice and 
in-service staff training 
which will increase staff 
competencies and knowledge 
of bilingual educational 
methodologies , program 
management , testing* and 
language training. 



To document and describe the 
nature and extent of preservice 
and in-service staff training. 



PARENT/COMMUNITY INVOLVEMENT 
AND EDUCATION . To hold reqular 
meetings and in-service sessions 
throughout the school year 
which stimulate pa rent /community 
participation in the program and 
increase their understanding of 
bilingual education approaches. 



To document and describe the 
nature and extent of parent/ 
community involvement and 
education opportunities. 
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EXHIBIT IV 



CORRESPONDING EVALUATION OBJECTIVE AND EVALUATION OUESTIONS 



SELECTED PRODUCT OBJECTIVE 



EVALUATION OBJECTIVE 



EVALUATION OUESTIONS 



SPEAKING AND UNDERSTANDING IN 
ENGLISH AS A SECOND LANGUAGE 
F OR LEP STUDENTS . To document 
a* * describe the extent to 
which LEP students in grades 
K-6 who receive individualized 
oral English language instruction 
improve their English oral pro- 
ficiency skills each program 
year relative to a comparison 
group that does not receive 
instruction. 



1. What is the nature and 
extent of the English 
oral language proficiency 
of LEP Students in the 
program and comparison * 
groups, at each K-6 grade 
level, prior to the pro- 
vision of program instruc- 
tion? 

2. What is the nature and 
extent of the English 
oral language proficiency 
of LEP students in the 
program and comparison 
groups, at each K-6 gx;ade 
level, at the end of each 
year of program instruction? 

3. How does the English oral 
language proficiency of LEP 
students in the program 
compare at the end of each 
program year with that of 
comparison students, and 

of English background peers? 

4. What is the nature and extent 

of the English language instruc- 
tion provided in each program 
classroom? 



5. How consistent is the nature 
and extent of the English 
language instruction pro- 
vided across program class- 
rooms? 
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6. How consistent is the nature 
and extent of the English 
languaye instruction provided 
with the description of the 
intended instructional model? 
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EXHIBIT IV (CONTINUED) 



SELECTED PROCESS OBJECTIVE 



t 



EVALUATION OBJECTIVE 



EVALUATION QUESTIONS 



STAFF DEVELOPMENT « To 
provide preservice and 
in-service staff training 
which will increase staff 
competencies and knowledge 
of bilingual educational 
methodologies, program 
management , testing, and 
language training. 



What were the topics and 
detailed content of pre- 
and In-service training 



provided to program staff? 



How were these topics decided 
upon? 



When were these sessions held? 



4. Who taught these sessions? 

5. What supplemental materials 
were provided at these sessions? 

6. Who attended these sessions? 

7. How did participants evaluate 
each of the sessions in terms 
of content, technique, and 
utility? 

8. What recommendations did 
participants make for improving 
or making useful extensions 

of staff development sessions 
for future program years? 

9- How did program management 
evaluate each of the sessions 
in terms of content, technique, 
and utility? 

10. What recommendations did 

program management make for 
improving or making useful 
extensions of staff development 
sessions in terms of content, 
technique, and utility? 
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EXHIBIT V 



EVALUATION 
OBJECTIVE & 
QUESTION 

_JS& 



DATA 
NEEDED 



DATA 
SOURCE 



DATA COLLECTION 
INSTRUMENT OR METHOD 



SPEAKING AND UNDERSTANDING 



IN ENGLISH AS A SECOND 



LANGUAGE FOR LEP STUDENTS 



to 

03 



What is the nature and 
extent of the English 
oral language proficiency 
of LEP students in the 
program and comparison 
groups, at each K-6 grade 
level, prior to the pro- 
vision of program instruc- 
tion? 

(1) What is the nature 
and extent of the English 
oral language proficiency 
of LEP students in the 
program and comparison 
groups,, at each K-6 grade 
level, prior to the pro- 
vision of program instruc- 
tion? 
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Measure of English 
oral language pro- 
ficiency prior to 
program instruction. 



LEP students at 
each K-6 grade 
in the program, 
and from the 
comparison group 
of LEP K-6 students 
which does not 
receive program 
instruction, but 
instead receives 
one of two dif- 
ferent English 
language instruc- 
tional approaches. 



Language Assessment Scale (LAS] 
I and II; Rating of bilingual 
certified teacher who is fluent 
in both languages and who has 
been trained in the procedure 
to rate English oral language 
proficiency. 
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(2) What is the nature and 
extent of the English oral 
language proficiency of LEP 
students in the program and 
comparison groups, at each 
K-6 grade level, at the end 
of each year of program 
instruction? 



Measure of English 
oral language pro- 
ficiency following 
program instruction 
each program year. 



(3) How does the Englis' Same as above for 

oral language proficiency questions (1) and 

of LEP students in the pro- (2) . 

gram compare at the end of 

each program year with that 

of comparison students, and 

of English background peers? 
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(4) What is the nature and 
extent of the English language 
instruction provided in each 
program classroom? 



Documentation of 
^"Ihdividualized 

instruction behaviors 
and conditions, time 
spent in English 
language instruction , 
and frequency of 
critical instructional 
variables. 



(5) How consistent is the Same as above. 

nature and extent of the 

English language instruction 

provided across program 

classrooms? 



(6) How consistent is the Same as above 

nature and extent of the 
English language instruc- 
tion provided with the 
description of the intended 
instructional model? 
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Same as above. 



Same as above. 
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Same as above for 
questions (1) and 
(2) , and -also from 
a group of English 
background peers. 



Same as above for 
(1) and (2). 



o 
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AduLts and condi- 
tions in program 
classroom. 



The School Survey and 
the Bilingual Reading 
Time Observation Schedule. 



Same as above. 



Same as above. 



Same as above, 
plus adults and 
conditions in 
comparison group 
classrooms. 



Same as above. 
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EXHIBIT VI 



HYPOTHETICAL 



LAS Oral Language Proficiency 
of LEP Program and Comparison Students 

(to be presented for all Grades, and also for each grade separately) 



Student group Pre-Test Score Post-Test Sc>re 

X SD X SD 



Program 

Comparison .v 

English-Speaking c ^ 

Peers <£p 



Tests of statistical significance (adjusting for change score concerns) would be run 
on this data, first for all grades combined; if a significant difference is found, 
tests would be run for each grade separately. 



Correlation of LAS and Teacher Rating 
of Oval Language Proficiency for LEP 
and Comparison Students 



Grade Preprogram Post program 
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EXHIBIT VII 

FORMAT FOR REPORTING INSTRUCTIONAL OBJECTIVES AND OUTCOMES 



Objective 5: Staff Development 

Objective . Staff competencies and knowledge of bilingual educational 
methodologies, program management, testing and language 
training will be developed through a program of preservice 
and in-service staff training. 

Intended Performance . A series of five activities will be implemented: 

5.1 Summer in-service program sponsored by Title VII Bilingual Con- 
sortium for Advancement; 

5.2 Career Ladder Program at the State University; 

5.3 District-sponsored Title VII workshops; 

5. A Ongoing workshop Provided by district support staff, Title VII 
Cross-cultural Resource "enter, and independent consultants; 

5.5 Classroom demonstration lessons for staff, provided by consultants 
and support staff. 



These activities will be documented and described by program staff, with parti- 
cipants providing evaluative information on each training occasion. 

Actual Performance , (A summary of the documentation and description in- 
formation with additional overview information of how participants felt 
about the efficacy of the training.) 

Program Issues . (A summary of critical concerns and events which were 
considered by either staff or participants to be problems of facilitating 
factors. Also discussed would be implications for changes for future 
staff development activities.) 



POST-TEST 



Part I s 

Establ ishing an Evaluation Team 

1. What are the major activities of an evaluation? 

2 What are some of the ways in which parents, community, and students 
may participate in a Title VII program evaluation? 

3. Name five important qualities that an evaluator should possess. 

4. Name five possible sources of information about potential external 
evaluator s. 

5. What is a common pitfall in kicking off a program evaluation? 

6. Describe two ways to establish initial agreements with a program 
evaluator . 

7. What are some of the key elements to be included in the final agree- 
ments reached with the external evaluator? 

Determining the Purposes and Objec t ives of the Evaluation 

1. Name five possible purposes of program staff for conducting a 
program evaluatior. 

2. in what two categories can evaluation objectives be placed? 
Describe the difference between these two basic approaches to 
program evaluation. 

Determining the Evaluation Questions, Variables, and Methodology. 

1 Describe the sequence of events following the establishment of the 
fluster of Evaluation objectives Which specify the elements 
of the evaluation design and process. 

2. Name and discuss two important considerations in the selection of 
language proficiency measures for the evaluation. 

3. What are some of the drawbacks of evaluation "burden" on staff and 
students? 

4. Typically, what will be contained in an evaluation report? 
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Part II: 



Gathering Evaluation Evidence 

1* What are some of the common concerns that may arise during the 
collection of evaluation evidence? 

Analysis of Evaluation Evidence 

2. How can program staff facilitate the preparation of evaluation 
evidence for analysis? 

Reporting the Evaluation 

1. What are the four basic types of reporting in an evaluation? 
Part III: 

1. What are the four basic things usually involved in using evalua^ 
results? 



POST-TEST ANSWERS 

Answers will be found on page 5, 



so 
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